<style type="text/css"> #Survey {
	width: 50%;
	margin: 0 auto;
}
</style>
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/3.4.1/css/bootstrap.min.css" />

<section class="container" id="Survey">
<h3> Instructions </h3>
<p>
We are building a virtual assistant that can help you control all your devices, social accounts, etc. 
Our virtual assistant can do a lot of things now but it still has a hard time to understand the users 
since different users may say the same thing very differently. So we need you to help us teach the assistant.
</p>
  <p>
In the following, you will see 4 different scenarios. In each scenario, you will see the command that was previously
  issued, how the assistant replied, and what you like to do next (in the form of a follow-up command). You will be asked to paraphrase the
  follow-up command in 2 different ways <span style="color:#FF0000;">without changing its meaning</span>, which means
<ul>
<li>Do not remove the device/service names if they exist in the original sentence</li>
<li>Do not change the meaning of the sentence even if doesn't seem to be useful</li>
<li>Do not drop any information unless it's obvious <b>without context</b> (beyond the context provided by the previously issued command);</li>
<li>Do not add any new information that doesn't exist in the sentence;</li>
<li><b>Keep quoted parts <span style="color:#FF0000;">(including the quote marks)</span>, emails, phone numbers identical</b>
<li><b>Do not drop @ signs and # signs</b>
</ul>
<span style="color:#FF0000;">Warning: for the last two requirements, we have a script to verify and if you failed to meet them, the script will reject your answer</span>
</p>

<p>Note that since all commands are generated by a computer, they sometimes may be unnatural and clunky.
  Your goal is to make the paraphrase as close as possible to the command you would use when interacting with the assistant.
Try to make your paraphrases natural, for example:</p>
<ul>
<li><b>Drop unnecessary components</b>, e.g., "send an email with some subject" -> "send an email" ("with some subject" does not give the virtual assistant any additional information)</li>
<li>Fix grammar mistakes</li>
</ul>

<p>Try your best to <font color="red"> make the 2 paraphrases as different as possible</font>.
If you really don't know how to paraphrase the command, if the context is not understandable, or if the follow-up is not meaningful in the given context, please write the exact words <samp>no idea</samp>.
</p>

<h3> Example 1 </h3>
<div class="panel panel-default">
<div class="panel-body">
<p><strong>Previous Command</strong></p>
<pre>
show me the stock price of Google</pre>
<p><i>The assistant shows the result</i></p>
<p><strong>Next Command</strong></p>
<pre>
notify me when it is below $1000</pre>
<p><strong>Correct paraphrases</strong></p>
<pre>send me a message when it goes under $1,000</pre>
<pre>alert me when the price is less than $1,000</pre>
  <pre>notify me when the stock price of Google goes below $1,000</pre>
<p><strong>Wrong paraphrases</strong></p>
<font color="red">incorrect information</font> <pre>notify me when the stock price of Yahoo goes below $1,000</pre>
  <font color="red">missing $ sign for "$1000"</font> <pre>notify me when it is below 1000</pre>
  <font color="red">wrong intent</font> <pre>when will it go below $1,000?</pre>
  <font color="red">wrong meaning</font> <pre>notify me when it is above $1,000</pre>
</div></div>

<h3> Example 2 </h3>
<div class="panel panel-default">
<div class="panel-body">
<p><strong>Previous Command</strong></p>
<pre>
show me youtube automotive channels</pre>
<p><i>The assistant shows the result</i></p>
<p><strong>Next Command</strong></p>
<pre>
how about cooking?</pre>
<p><strong>Correct paraphrases</strong></p>
<pre>how about cooking channels?</pre>
<pre>search cooking channels instead</pre>
<pre>search recipes videos instead</pre>
<p><strong>Wrong paraphrases</strong></p>
  <font color="red">departs from context</font> <pre>search cooking instead</pre>
<p><strong>Acceptable but discouraged paraphrases</strong></p>
<font color="red">repeats previous command</font> <pre>how about automotive channels?</pre>
</div></div>

<h3> Example 3 </h3>
<div class="panel panel-default">
<div class="panel-body">
<p><strong>Previous Command</strong></p>
<pre>
post a picture on twitter</pre>
<p><i>The assistant confirms the command before executing</i></p>
<p><strong>Next Command</strong></p>
<pre>
do this automatically when I upload a picture to instagram</pre>
<p><strong>Correct paraphrases</strong></p>
<pre>do it when I post to Instagram</pre>
<pre>post it when I upload to Instagram</pre>
<pre>not now, but when I upload to Instagram</pre>
<pre>do it for every new picture I have on Instagram</pre>
<p><strong>Wrong paraphrases</strong></p>
  <font color="red">missing information (Instagram)</font> <pre>do it when I upload a picture</pre>
  <font color="red">wrong meaning</font> <pre>upload a picture to Instagram</pre>
<p><strong>Acceptable but discouraged paraphrases</strong></p>
<font color="red">repeats previous command</font> <pre>post a picture on twitter when I upload to Instagram</pre>
</div></div>

<h3> Your turn now </h3>
<div class="panel panel-default">
<div class="panel-body">
<p><strong>Previous Command</strong></p>
<pre>
${context_utterance1}</pre>
<p><i>${assistant_action1}</i></p>

<p><strong>Next Command</strong></p>
<pre>
${sentence1}</pre>
<label for="Paraphrase1-1">Paraphrase 1</label>
<div class="form-group"><textarea class="form-control" cols="120" how="" id="Paraphrase1-1" name="Paraphrase1-1" required="" rows="3"></textarea></div>
<label for="Paraphrase1-2">Paraphrase 2</label>
<div class="form-group"><textarea class="form-control" cols="120" how="" id="Paraphrase1-2" name="Paraphrase1-2" required="" rows="3"></textarea></div>
</div>
</div>
<br>

<div class="panel panel-default">
<div class="panel-body">
<p><strong>Previous Command</strong></p>
<pre>
${context_utterance2}</pre>
<p><i>${assistant_action2}</i></p>

<p><strong>Next Command</strong></p>
<pre>
${sentence2}</pre>
<label for="Paraphrase2-1">Paraphrase 1</label>
<div class="form-group"><textarea class="form-control" cols="120" how="" id="Paraphrase2-1" name="Paraphrase2-1" required="" rows="3"></textarea></div>
<label for="Paraphrase2-2">Paraphrase 2</label>
<div class="form-group"><textarea class="form-control" cols="120" how="" id="Paraphrase2-2" name="Paraphrase2-2" required="" rows="3"></textarea></div>
</div>
</div>
<br>

<div class="panel panel-default">
<div class="panel-body">
<p><strong>Previous Command</strong></p>
<pre>
${context_utterance3}</pre>
<p><i>${assistant_action3}</i></p>

<p><strong>Next Command</strong></p>
<pre>
${sentence3}</pre>
<label for="Paraphrase3-1">Paraphrase 1</label>
<div class="form-group"><textarea class="form-control" cols="120" how="" id="Paraphrase3-1" name="Paraphrase3-1" required="" rows="3"></textarea></div>
<label for="Paraphrase3-2">Paraphrase 2</label>
<div class="form-group"><textarea class="form-control" cols="120" how="" id="Paraphrase3-2" name="Paraphrase3-2" required="" rows="3"></textarea></div>
</div>
</div>
<br>

<div class="panel panel-default">
<div class="panel-body">
<p><strong>Previous Command</strong></p>
<pre>
${context_utterance4}</pre>
<p><i>${assistant_action4}</i></p>

<p><strong>Next Command</strong></p>
<pre>
${sentence4}</pre>
<label for="Paraphrase4-1">Paraphrase 1</label>
<div class="form-group"><textarea class="form-control" cols="120" how="" id="Paraphrase4-1" name="Paraphrase4-1" required="" rows="3"></textarea></div>
<label for="Paraphrase4-2">Paraphrase 2</label>
<div class="form-group"><textarea class="form-control" cols="120" how="" id="Paraphrase4-2" name="Paraphrase4-2" required="" rows="3"></textarea></div>
</div>
</div>


</section>
